Software programs that are designed for execution on a microcontroller are typically validated to ensure that the microcontroller functions correctly while executing the program. The validation can be conducted against a “test case,” or a test “suite” of several test cases. A test case is a scripted, automated use of the program, typically in a test environment (as opposed to a production environment) that emulates the execution on the microcontroller, to validate the program's performance of a particular task. A test suite is a collection of test cases that together test all of the program's functions being validated.
A program developer may be asked or required to show that its product test suites “cover” (i.e., perform validation tests on) a certain percentage of the executable code in a project; this percentage is often 100%. Validating the executable code requires analyzing the behavior of the system via the system's assembly instructions. Embedded systems within microcontrollers can include thousands or even millions of assembly instructions.
An early historical method of determining code coverage was to track every instruction execution of a selection of code, and tabulate which instructions had been executed. This method typically requires simulators and hardware emulators with sufficient computing power and data storage to track every instruction execution, of which there may be millions.
A modern extension of this technique is available on (usually larger) microcontrollers equipped with an “instruction trace” mechanism. This mechanism typically requires expensive, high-speed hardware to capture the trace stream and exorbitant amounts of processing time by an external program (usually an IDE) to store, then retrieve and analyze what could easily be many gigabytes (or more) of trace data, which may be both expensive and time consuming. In addition, the correlation of instruction to “high-level language statement” is tenuous when the source code was written in a high-level language (like C) rather than assembly language. Although there is often a correlation of instructions to statements, it is not always easy to determine, and is rarely fixed from run-to-run. For example, if compiler optimization settings have changed, the flow of instructions may be dramatically different, so any correlation must be dynamic and cognizant of many changes in the executable image build and execution environment. This approach has the benefit of being accurate at a very fine granularity, but it is extremely inefficient for large bodies of code or long, involved test runs.
A subsequent approach was also derived from theory embodied in the development of high-level languages. This approach entails instrumenting branch points—known as “basic blocks”—with code that transmits a text message via a serial communication port. The idea of a “basic block” derives from a contiguous set of machine instructions with a single entry point. The boundaries of a basic block are easily determined by the compiler, but difficult to determine precisely from generated code. This led to the refinement of the process discussed above. By determining that a basic block has been executed, the system knows that every statement in the basic block has been executed. This leads to a reduction of the best case number of events to be tracked. Unfortunately, a non-trivial program typically contains a large number (e.g., hundreds) of basic blocks, which may require a substantial or unavailable amount of resources (e.g., memory) to tabulate their execution. Therefore, practical examples of this method have resorted to instrumenting source code with large and time consuming communications of an identification of the basic block being executed. On some microcontrollers, this can take the form of a call to a routine which outputs the basic block identification, along with a viable “envelope,” over an available communications channel, often a UART. This approach has three significant drawbacks: 1) the instrumentation code can significantly impact the timing characteristics and performance of the code under test, 2) the instrumentation code is fairly large and may require only parts of the application to be instrumented in each test run, which has a multiplicative effect on the total time required to gather data on the code coverage, and 3) collecting data over partial runs introduces the unwanted effect of environmental and other test factors not being precisely the same from run to run.
Another known approach involves the use of simulators that are designed to track the execution of every instruction and are thus able to track which instructions are executed in situ. However, a programmer's customer might not accept simulated code coverage based on the view that a simulator is not sufficiently representative of the actual device.